The E-Commerce Recommendation Data Mart is a scalable AWS-based data platform designed to power recommendation engines with clean, analytics-ready interaction data. It ingests clickstream, product, and transaction logs, transforms them via AWS Glue (PySpark) into a user–product interaction matrix, stores curated data in an S3 Lakehouse with partitioning, and prepares features for Amazon SageMaker recommendation models.
Key outcomes:
• Handles up to 1 TB/day of e-commerce logs
• Optimized querying via Athena on partitioned S3 data
• Ready-to-use feature store for ML-driven recommendations
• End-to-end ETL + lakehouse warehousing for cost-effective analytics
• Delivered in ~8 months, on time, with scalability and governance built in
This foundation enables personalized recommendations, improving user engagement and sales.
The solution uses a serverless, lakehouse-style architecture on AWS:
Data Sources: Clickstream logs (web/app), product catalog exports, transaction data
Ingestion Layer: Raw data lands in S3 (optionally via Kinesis for streaming)
Processing Layer: AWS Glue ETL in PySpark builds the user–product interaction matrix and feature sets
Storage Layer: Amazon S3 Lakehouse with Hive-style partitioning (e.g., by event_date)
Query Layer: Athena for SQL analytics on curated parquet data
ML Layer: SageMaker Feature Store (optional) for model training/serving
This architecture is fully cloud-native, scalable, and cost-efficient.
Raw Layer (S3): Buckets like raw-logs/ store clickstream, product, and transaction data (JSON/CSV).
Transformed / Data Mart Layer (S3): transformed-matrix/ and data-mart/ store: User–product interaction matrix (views, adds-to-cart, purchases) and Aggregated interaction scores per user_id / product_id. Partitioned by event_date (and optionally user segments) for performance.
Athena External Tables: Defined over S3 Parquet data with Glue Data Catalog + MSCK REPAIR TABLE for partitions. Acts as the analytical layer for BI / ad-hoc queries.
Feature Store Layer (Optional): SageMaker Feature Groups for user, item, and interaction features, backed by S3.
Extract: Logs from clickstream trackers, product catalogs, and transaction systems are exported to S3 (optionally via Kinesis → S3). Glue crawlers or S3 event notifications detect new files.
Transform: PySpark ETL jobs: Clean and standardize input data. Aggregate interactions by user_id and product_id (views, add-to-cart, purchases). Build a user–product interaction matrix and other features (implicit feedback style).
Load: Write partitioned Parquet datasets to S3 (Snappy-compressed). Register/update Athena external tables. Optionally push curated interaction data into SageMaker Feature Store for ML. The pipeline uses Glue job bookmarks for incremental loads and can run in batch or near real-time.
Project Start: March 1, 2025 | Deployment Complete: November 13, 2025 | Total Duration: ~8 months
Testing: Unit tests (PyTest) for ETL logic with mocked DataFrames. Integration tests: end-to-end runs with sample logs. Performance tests: query benchmarks in Athena with < 10s response for daily aggregates. Data validation: matrix values and aggregates cross-checked against raw logs with ~99% match.
Deployment: Terraform used for infrastructure-as-code (S3, Glue, Athena, IAM). Glue jobs scheduled via EventBridge. Rollout to production with controlled schedules and logging enabled.
Monitoring: CloudWatch metrics and alarms on Glue job runtime, failures, and retries. S3 storage and Athena usage monitored with AWS cost tools.
Maintenance: Automated backups via S3 versioning. Monthly cost reviews and optimization (e.g., compression, partition tuning). Security via encryption at rest, least-privilege IAM roles, and parameterized configs (SSM / job arguments).
Estimated Cost: ~$300/month for moderate scale (Glue + Athena + S3), depending on query and ETL frequency.
Methodology: Agile with 2-week sprints covering design, ingestion, ETL, warehousing, feature store, and testing phases. Risks such as data skew and large-scale partitions were handled via dynamic allocation and partition strategy tuning.
.png)
.png)
.png)
.png)
.png)